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[57] ABSTRACT 

A method and apparatus for appropriately and without error 
extracting specific feature areas of a facial image, and 
outputting accurate coordinate data for the extracted facial 
features for use in such applications as three-dimensional 
morphing, image blending, or facial image identification. 
Preferably, the inclusion of a searching range setting unit for 
each facial feature to be extracted develop a searching range 
for extracting each facial feature referenced to individual 
points specified by the user for the mouth and left and right 
eyes based on the coordinate data of those specified points. 
The areas of those facial features are then extracted from 
within the defined searching ranges by area extractors, and 
the coordinate data of the contour points forming the con- 
tours of those areas is extracted by contour point extractors. 
Based on the coordinate values of the specific point specified 
for each facial feature area, extraction content evaluators 
then evaluate the relative positions of the extracted areas to 
determine whether said areas were correctly extracted. 

56 Claims, 17 Drawing Sheets 
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FACIAL IMAGE PROCESSING METHOD OBJECTS OF THE INVENTION 

AND APPARATUS Jt fa mere f ore m ob j ect 0 f tne presen t invention to greatly 

BACKGROUND OF THE INVENTION reduce the processing time required for facial feature 

1. Field of the Invention extraction, enable accurate facial feature extraction with 
The present invention generally relates to facial image 5 minimal errors, and reliably extract the contour points of the 

processing techniques, and more specifically, an improved extracted individual feature areas by setting searching areas 

facial image processing method and apparatus for generating having a defined, limited range, and extracting the facial 

feature coordinate information corresponding to character- feature areas only within each of the set searching areas, 

istic parts of a facial image useful in facial morphing, ^ It is a further object of the present invention to enable high 

identification or blending operations. precision extraction of facial feature areas by a post pro- 

2. Description of the Related Art cessing function for evaluating the extracted content. 
To merge two facial images, it is first necessary to extract SUMMARY OF THE INVENTION 

the eyes, eyebrows, mouth, facial contours, and other facial 

features, calculate the coordinate data of the representative 15 In accordance with these and related objects, image 

contour points forming the contours of each of the extracted processing according to the present invention comprises 1) 

features based on the extracted image data, and then transfer preliminary or rough image feature identification based on 

the calculated coordinate data to the image-merging proces- analysis of the scanned subject image or through acceptance 

sor in order to match a wire frame model of the facial shape of user identified specific feature points; 2) procurement of 

to the actual facial image. 2 o " na 8 c ^arch ranges for each image feature identified with 

One facial feature extraction method commonly used me user-supplied specific feature points in order to generate 

detects the image edges based on the coordinate data and accurate image search ranges, where the boundaries and 

black/white density data of the input image data, and digi- characteristic identifiers of a particular search range depends 

tizes the image edges to extract the facial features. With this 011 the classification of the image feature to which it relates; 

method, however, much noise is introduced into the digital 2S and 3) characteristic extraction for each image feature based 

image during processing, and processing of images rich in on analyzing the image portions defined by the procured 

color information is limited. To resolve these problems, search ranges. Preferably, in the case of facial image feature 

recently developed methods attempt to distinguish and extraction, the user will select one or more points on a 

extract the facial features using the color information con- digitized image, each corresponding to at least one facial 

tained in the image 30 feature (e.g. left and right eyes, nose and mouth) in which 

One such method follows a predetermined program to characteristic information is desired for further processing or 

process and extract the facial features based on the coordi- identification purposes. 

nate and color data of the input image data. This method first Facial image processing according to the present pre- 

stores the input image data to an image data memory, and ferred embodiments of the invention further involves having 

then extract the facial features in a predetermined sequence, 35 a USCT P° mts at least roughly approximating the 

e.g., the mouth (i.e., lips) first, followed by the eyes, the location of the right and left eyes and mouth of the facial 

eyebrows, and other facial parts, according to a predefined image. Then, search ranges are determined for the mouth, 

program. To extract the mouth, for example, the complete right and left eyes, eyebrows, and general facial contour, 

facial image data is queried to locate the image area using Further, preferably, the searching range for the mouth area 

the YIQ color system, where the Q component, which 40 may be set in the relative x-axis and y-axis directions to 

expresses lip color, exceeds a certain threshold value. This include the mouth area referenced to the point specified as 

area is extracted as the mouth area, which is then processed the mouth based on the difference between the x-coordinate 

to obtain and output the coordinate data for the representa- data of the points specified as the right and left eyes, wherein 

tive contour points forming the contour of the mouth and the x-coordinate axis represents the side-to-side direction of 

circumscribing lips. The coordinate data for the eyes and 45 the facial image and the y-coordinate axis represents the 

eyebrows is processed in the same manner to extract the vertical direction of the facial image. Alternatively, the 

respective areas and obtain the corresponding coordinate searching range for the mouth area may be set in the x-axis 

data. and y-axis directions to include the mouth area referenced to 

As described above, these refined color techniques follow the point specified as the mouth based on the difference 

a predefined program to extract the facial feature areas from 50 between the x-coordinate data of the points specified as the 

the complete image data, and to obtain the coordinate values right and left eyes, and the average of the differences 

for the extracted feature areas. In the mouth extraction between the y-axis coordinate of the point specified as the 

process, for example, the collection of image areas in which mouth and the y-axis coordinates of the points specified as 

the Q component exceeds a known threshold value is the right and left eyes. Again, x-coordinate axis here repre- 

determined to be the mouth, and the mouth area is thus 55 sents the side-to-side direction of the facial image and the 

extracted based on the value of the Q components. However, y-coordinate axis represents the vertical direction of the 

when there is a Q component (red) area in the background facial image. 

of the facial image exceeding a predetermined saturation or Moreover, the searching ranges for the eyes may be set in 

level size, or when there is a red pattern exceeding a the x-axis and y-axis directions to include the eyes refer- 

predetermined level in the subject's clothing, these areas 60 enced to the points specified as the right and left eyes based 

may be mistakenly determined to be the area of a mouth. on the difference between the x-axis coordinate data of the 

Naturally, this results in an inaccurate feature characteriza- points specified as the right and left eyes. Likewise, the 

tion of the input facial image. Significant processing time is searching ranges for the eyebrows are set in the x-axis and 

also required to extract plural facial features, including the y-axis directions to include the eyebrows referenced to the 

eyes, eyebrows, and mouth, from the complete facial image 65 points specified as the right and left eyes based on the 

data, particularly in 16 bit high color or 24 bit true color difference between the x-axis coordinate data of the points 

applications. specified as the right and left eyes. 
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Also, preferably, the searching ranges for the facial con- direction of the facial feature image regions; 4) definition of 

tours in the areas at the eye height in the y-axis direction may a searching range with a width in the x-axis direction equal 

be set in the relative x-axis and y-axis directions to include to a known Vn fraction of the distance between the right and 

the facial contours referenced to the points specified as the left end points for each respective facial feature area in each 

right and left eyes based on the average of the differences 5 area near the midpoint between its corresponding right and 

between the y-axis coordinate of the point specified as the left end points; and finally 5) determination of the coordi- 

mouth and the y-axis coordinates of the points specified as nates of the top and bottom end points in the y-axis direction 

the right and left eyes, where the x-coordinate axis repre- of the eye or mouth facial feature areas within their respec- 

sents the side-to-side direction of the facial image and the tive defined searching ranges. Moreover, the coordinates of 

y-coordinate axis represents the vertical direction of the 1Q the contour points between the obtained four right, left, top, 

facial image. Plus, the searching ranges for the facial con- and bottom end points are obtained with the x-axis coordi- 

tours in the areas at the mouth height in the y-axis direction nate thereof calculated as the average of the x-axis coordi- 

may be set in the x-axis and y-axis directions to include the nates of the contour points on either side of the contour point 

facial contours referenced to the point specified as the mouth being calculated. 

based on the average of the differences between the y-axis 15 Also, facial image processing according to this embodi- 

coordinate of the point specified as the mouth and the y-axis ment may further comprise image area verification through 

coordinates of the points specified as the right and left eyes. comparison of the image positions of the extracted areas 

Finally, the searching range for the facial contour in the area with respect to the coordinate values of a specific point in 

of the mouth in the x-axis direction may be set in the x-axis each facial feature area to determine whether such facial 

and y-axis directions to include said facial contour refer- 2Q feature areas were correctly extracted. When the extracted 

enced to the point specified as the mouth based on the area is the mouth area, preferably this verification step or 

average of the differences between the y-axis coordinate of apparatus involves determining whether a line connecting 

the point specified as the mouth and the y-axis coordinates the right and left end points in the x-axis direction of the 

of the points specified as the right and left eyes. mouth area is sloped greater than a predetermined angle 

To improve characteristic feature extraction accuracy, 25 relative to a line through a specific point in each of the right 

facial image processing according to the presently preferred and left eyes. Also, when the extracted area is a facial 

embodiments may comprise, in addition to preliminary contour area, the plural contour points forming the facial 

image feature identification, image search range develop- contour may be processed in units of three adjacent contour 

ment and feature characteristic extraction, the evaluation of points; the coordinates of the middle of each three contour 

the relative positions of the extracted feature images based 30 points are compared with the average of the coordinates of 

on the coordinate values of the previously supplied or the two adjacent contour points; and the average of the 

determined specific points in the facial feature areas to coordinates of the two adjacent contour points are substi- 

determine whether the facial features were, in fact, correctly tuted for the coordinates of the middle contour point if the 

extracted. position of the middle contour point is determined to be to 

In the context of the mouth area of a scanned facial image, 35 the inside of the face relative to the adjacent contour points 

the aforementioned evaluation device or process comprises: of the subject facial image. 

comparison of the y-axis coordinates of the right and left end Based on the coordinates of the three points specified by 

points in the x-axis direction of the extracted mouth area the user for the right and left eyes and the mouth, the 

with the y-axis coordinate of the midpoint in the x-axis techniques of the present invention set a separate searching 

direction at the top edge of the extracted mouth area. When 40 range for the mouth and each of the eyes, and search for the 

the y-axis coordinate of the midpoint in the x-axis direction respective facial features only within those limited searching 

is smaller than the y-axis coordinate of either right or left ranges. As a result, processing is greatly simplified when 

end point in the x-axis direction of this mouth area, the compared with methods extracting facial features from the 

extracted mouth area is determined to represent an open entire facial image, and the processing time can therefore be 

mouth, the extracted mouth image is determined to include 45 greatly reduced. Furthermore, because the respective facial 

the lower lip only, and a searching range for additionally features are extracted from within defined image areas, false 

extracting the upper lip is therefore set. extraction of other facial features in place of the desired 

In addition, evaluation may further preferably comprise facial features rarely occurs, and the facial feature the user 

determination whether the eyes or mouth were correctly wishes to extract can be reliably extracted, 

extracted by determining whether the respective points 50 In addition, after extracting the areas of the facial features, 

specified by the user as the center of the eyes or mouth are the contour points of the eyes and mouth may be extracted 

contained within the extracted areas. Also, evaluation could from the extracted feature areas, the coordinates of the right 

include verification of correct eyebrow locations by evalu- and left end points in the x-axis direction of the eye or mouth 

ating the positions of the coordinates of the right and left end image may be obtained, and then the coordinates of the top 

points of the extracted eyebrow area relative to the coordi- 55 and bottom end points in the y-axis direction can be 

nates of the point specified as the adjacent eye. obtained. Before obtaining the coordinates of the top and 

Alternatively, facial image processing according to the bottom end points in the y-axis direction, however, a search- 
present invention may comprise: 1) extraction of image ing range with a width in the x-axis direction equal to a 
regions or areas corresponding to each facial feature using a known Vn fraction of the distance between the right and left 
coordinate system, wherein the x-coordinate axis represents 60 end points should be set, and the top and bottom end points 
the side-to-side direction of the facial image and the in the y-axis direction are obtained within this searching 
y-coordinate axis represents the vertical direction of the range. As a result, the top and bottom contour points at the 
facial image; 2) procurement of the contour points forming highest and lowest points representative of the image can be 
the contours of the eyes and mouth in the corresponding obtained reliably without scanning and digitization noise 
facial feature areas extracted during facial image region 65 effects. 

extraction; 3) development, from these contour points the After the areas of the desired facial features are extracted, 

coordinates of the right and left end points in the x-axis the relative positions of the extracted areas may be evaluated 
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based on the coordinates of specific points in each of the 
facial feature areas. It is therefore possible to check whether 
the facial features were correctly extracted and to automati- 
cally reprocess the image when the feature extraction 
appears wrong when evaluating these points, and errors in 
the extraction process can therefore be significantly reduced. 

Other objects and attainments together with a fuller 
understanding of the invention will become apparent and 
appreciated by referring to the following description of 
specific preferred embodiments and claims taken in con- 
junction with the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

In the drawing?, wherein like reference symbols refer to 
like parts, 

FIG. 1 is a block diagram used to describe the first 
embodiment of the present invention; 

FIG. 2 graphically depicts an input facial image; 

FIG. 3 is used to describe setting the mouth and eye 
searching ranges according to the first embodiment of the 
invention; 

FIG. 4 is used to describe an alternative technique for 
setting the mouth searching range according the first 
embodiment of the invention; 

FIG. 5 is used to describe setting the eyebrow searching 
range according to the first embodiment of the invention; 

FIG. 6 is used to describe setting the mouth searching 
range according to the first embodiment of the invention 
when the mouth is open; 

FIG. 7 is used to describe setting the facial contour 
searching range according to the first embodiment of the 
invention; 

FIGS. 8A and 8B are used to describe an example of 
evaluating the extracted content according to the first 
embodiment of the invention; 

FIG. 9 is a simple block diagram used to describe the 
second embodiment of the present invention; 

FIG. 10 is used to describe extracting the contour points 
forming the eye contour according to the second embodi- 
ment of the invention; 

FIG. 11 is used to describe extracting the contour points 
forming the mouth contour according to the second embodi- 
ment of the invention; 

FIG. 12 is used to describe extracting the contour points 
forming the eye contour according to the second embodi- 
ment of the invention when the eye is on an angle; 

FIG. 13 is used to describe evaluating the extracted 
content of the mouth image according to the second embodi- 
ment of the invention; 

FIG. 14 is used to describe the method of evaluating the 
extracted content of the facial contour image in the second 
embodiment of the invention; 

FIG. 15 is a representative processing system incorporat- 
ing the teachings of the first embodiment of the invention; 

FIG. 16 is a flowchart illustrating mouth area range setting 
according to the first embodiment as described with refer- 
ence to FIG. 3; 

FIGS. 17 and 18 are flowcharts depicting mouth area 
range setting according to the first embodiment as described 
with reference to FIG. 4; 

FIG. 19 is a flowchart depicting eye area range setting 
according to the first embodiment as described with refer- 
ence to FIG. 3; 
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FIG. 20 is a flowchart illustrating mouth extraction veri- 
fication according to the first embodiment as described with 
reference to FIG. 6; 

FIG. 21 is a flowchart illustrating facial contour area 
5 range setting according to the first embodiment as described 
with reference to FIG. 7; 

FIG. 22 is a flowchart illustrating eyebrow extraction 
verification according to the first embodiment as described 
with reference to FIGS. 8A and 8B; and 

FIG. 23 is a flowchart illustrating eye contour extraction 
according to the second embodiment. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

15 

The preferred embodiments of the present invention are 
described below with reference to the accompanying figures. 
First Embodiment 
FIG. 1 is a block diagram used to describe significant 

20 aspects of first embodiment of the present invention. 
Overall, this embodiment comprises, as shown in FIG. 1, 
image memory 2 for storing the input image data preferably 
contained in system memory 220 (not shown, See FIG. 15); 
feature unit searching range setting units 3 for setting the 

25 searching ranges for the facial features to be extracted (i.e., 
the right and left eyes, eyebrows, mouth, and facial contours 
in this embodiment based on the contents of image memory 
area 2 and memory segments 100a, 100£> & 100c (not 
shown, see FIG. 15), each holding user defined or calculated 

30 center point values for the eyes and mouth for the digitized 
facial image contained in image memory area 2); feature 
unit extractor 4 for extracting the area of the respective 
feature units in the set searching ranges; contour point 
extractor 5 for extracting the representative contour points 

35 forming the feature unit contours from the extracted feature 
unit areas, and outputting the corresponding coordinate data 
to coordinate data memory area 110; and extracted content 
evaluator 6 for evaluating whether the extracted content is 
correct or not correct based on the coordinate data. 

40 According to the presently preferred embodiments, a 
multiple tone color facial image or similar feature-rich 
image is preferably scanned in through the use of image 
input member 140 (FIG. 15). A simplified, black and white 
example of such an image is shown in FIG. 2. The scanned 

45 image is then digitized into corresponding color grade image 
data at scanner interface 130, with the assistance of CPU 120 
and system memory 220 of computer processing system 
200. Then, the composite scanned digitized image is depos- 
ited into the image memory area 2 of system memory 220 in 

50 a known format. It should be recognized here that the subject 
facial image as depicted in FIG. 2 need not be necessarily 
originally digitized by the processing system of FIG. 15, and 
can instead be already accessible to processing system 200 
through the external store 180, data storage device 150/ 

55 removable media 160, or system memory 220 itself. 

Ibming back to FIG. 1, feature area searching range 
setting unit 3 preferably comprises a searching range setting 
subunit for each of the feature units to be extracted, i.e., a 
mouth searching range setting unit 31, eyes searching range 

60 setting unit 32, eyebrows searching range setting unit 33, 
and facial contour searching range setting unit 34 for the 
mouth, right and left eyes, eyebrows, and facial contour 
features units extracted for this embodiment. 
Feature unit extractor 4 similarly comprises correspond - 

65 ing mouth area extractor 41, eye area extractor 42, eyebrow 
area extractor 43, and facial contour extractor 44 for extract- 
ing the feature units within their respective searching ranges. 



07/09/2004, EAST Version: 1.4.1 



5,933,527 

7 8 

Contour point extractor 5 likewise comprises a mouth contour point extractor 5 including contour point extractor 
contour point extractor 51, eye contour point extractor 52, subunits 51-54; and extracted content evaluator 6 including 
eyebrow contour point extractor 53, and facial contour point extraction content evaluator subunits 61-64; all may par- 
extractor 54 for extracting the representative contour points tially or fully reside within external store 180, removable 
forming the contours of the corresponding areas based on 5 media 160 and/or system memory 220 as is well known in 
the respective output from the feature unit area extractors the art. 

41-44, and extracting the coordinate data of the extracted Moreover, image memory 2 and use defined feature point 

contour points useful in blending, morphing, image identi- registers 100a, 100f> and 100c may be constructed within 

fication or similar image processing applications in which system memory 220 as integral part thereof or may comprise 

three dimensional aspects and locations of images are evalu- 10 discrete circuitry as well known in the computing arts, 

ated and/or manipulated. Still referring to FIG. 15, removable media 160 may 

Also, extracted content evaluator 6 likewise comprises include a floppy disk, CD ROM, ROM cartridge or other 

mouth extraction content evaluator 61, eye extraction con- similar apparatus for retaining information in computer 

tent evaluator 62, eyebrow extraction content evaluator 63, readable form. Similarly, external store 180 may include 

and facial contour extraction content evaluator 64 for deter- 15 another processing system, a computer readable storage 

mining whether the content extracted by the contour point component a collection or network of processing systems 

extractors 51-54 is correct. and/or storage components, or similar device or devices in 

It should be noted here that components of this invention communication with processing system 200 to exchange 

may be conveniently implemented using a conventional information including the above-mentioned read instruc- 

general purpose digital computer programmed according to 20 tions. Further, in these embodiments, processing system 200 

the teachings of the specification, as will be apparent to is indicated as being a general purpose digital computer, 

those skilled in the computer arts. Appropriate software However, inordinary skill with knowledge of this disclosure 

coding can be readily prepared based on the teachings of the will understand that the particular processing system could 

present disclosure, as will be apparent to those skilled in the alternatively include, for example, a special purpose dedi- 

software art. The present invention may also be imple- 25 cated micro-controlled subsystem or similar processing 

mented by preparation application specific integrated cir- device as long as it has sufficient resources for at least 

cuits (ASICs) or by interconnecting an appropriate network sequential execution of the techniques described hereinbe- 

of conventional component devices and circuits, as will be low. 

readily apparent to those skilled in the art. Also shown in FIG. 15 is printer 195 communicating with 

In each of the following preferred embodiments of the 30 system bus 210 and CPU 120 through printer interface 190. 

present invention, including but not limited to, the first In this embodiment, printer 195 may be used as a hard copy 

embodiment as shown in FIG. 1, the feature area searching output device for coordinate data and feature extraction 

range setting unit 3, including specific feature searching information developed by the interdependent threads 3 

range setting units 31-34; feature unit extractor 4, including through 6 and the results and coordinate data information 

specific feature area extractors 41-44; contour point extrac- 35 110 contained within system memory 220. Likewise, display 

tor 5, including specific feature contour point subextractors 240 is used to impart graphical images including a likeness 

51-54; and extracted content evaluator 6, including specific of the subject facial images shown in FIG. 2 to an end user 

facial feature extraction content subevalutors 61-64, will all of processing system 200. As shown in processing system 

take the form of interdependent threads executing on a 200, CPU 120 directs image data to be displayed on display 

general purpose processing system 200 (FIG. 15). These 40 240 by communicating over system bus 210 to I/O interface 

threads permit the processing system 200 to carry out the 230 which translates the image data signals into appropriate 

image development techniques and processes of the present CRT RGB values for imparting a two dimensional rendition 

invention when the processing system 200 reads and of image to the end user. A mouse 250 is also shown 

executes their corresponding programming instruction from connected to 1/0 interface 230 of computer system 200 for 

a computer readable storage media. The storage medium 45 receiving user input based in part in reaction to image 

containing the thread instructions can include, but is not information presented to the user on display 240. 

limited to, any type of disk media including floppy disks, Alternatively, as is well known in the art, other input devices 

optical disks, CD-ROMS, magneto-optical disks, hard may be used to capture user input including but not limited 

drives or disk arrays, whether located within or external to to voice processing equipment, a keyboard, a touch pad, etc. 

the processing system 200. Alternatively, the storage 50 It should also be noted here that the points specified for 

medium can include ROM, RAM, EPROM, EEPROM, the three feature units processed, i.e. the mouth and the right 

FLASH EEPROM or any other type of storage medium and left eyes (ideally, the center point of the mouth and the 

suitable for at least temporary storage of computer readable center points of the right and left eyes) are selected by the 

instructions. user in the present embodiment, and the searching ranges for 

Turning now briefly to FIG. 15, FIG. 15 illustrates in more 55 the mouth, eyes, and eyebrows are set based on the coordi- 

detail the representative processing system 200, which nates of these user-specified points. Tnis is described below, 

includes system memory 220, CPU 120 and scanner inter- For example, the user may use the mouse 250 to specify 

face 130 interconnected via system bus 210 well known in the center points pi and p2 of the two eyes el and e2, and 

the computer arts. Also connected to the system bus 210 is the center point p3 of mouth m in a facial image as shown 

system addressable storage device 150 capable accepting, 60 in FIG. 2 and on the display 240. It is assumed below that 

reading and writing information to and from a type of the relative (x, y) Cartesian coordinates of these three 

removable media 160 and external store 180 illustrating designated points pi, p2, and p3 are, respectively, (xl, yl), 

storage mediums in communication with processing system (x2, y2), and (x3, y3). 

200. Accordingly, in this processing system, programming The center points having been specified, mouth searching 

instructions corresponding to the feature area searching area 65 range setting unit 31 executes the following process, 

range setting unit or its respective subunits 31-34, feature Control begins at step S1000 as shown in FIG. 16. Control 

unit extractor 4 including area extractor subunits 41-44; thereafter passes the step S1010, in which the mouth search- 
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ing range setting unit 31 queries the image data and user 

defined points for the mouth and eyes chosen by the user. r -—)u> (y3 + — 1 

Control passes to step S1020 in which the mouth searching I 2 J V 2 J 

range setting unit obtain the difference dx between the x 

coordinates of the centers points pi and p2 left and right 5 m ^ ^ ^ ^ defined u ^ 

eyes el and e2 respectively (dx=x2-xl) to define the mouth for m Control then tenninates Qa ^ 

searching range in the x axis direction based on the center siHO 

point of the mouth (user defined point p3), as shown in FIG. ' . 

3. Hereafter, control passes to step S1030, in which the Meanwhde, eyes searching range setting unit 32 then 

mouth area search range in the x and y directions is 10 determines the searching ranges for the eyes el and e2. The 

established. More particularly, in this embodiment, one-half method a PP lied here 15 the «"* , for b ° m 50 th , e 

0 £ ^ following description is therefore limited to left eye el. 

Referring again to FIG. 3 and now the flow chart of FIG. 19, 
^ control with the eye searching range setting unit 32 begins 

y J 15 at step S1150. Control thereafter passes to step S1160 in 

which the image data and the image defined points for the 
mouth and eyes are acquired as described hereinabove, 
is obtained to define the mouth searching range in the y-axis Thereafter, at step S1170, the horizontal difference between 
direction, and is based on the user-defined center point p3 of 2Q the eyes is calculated as discussed hereinabove with refer- 
the mouth. Controlled and terminates normally in step ence to the mouth searching range setting unit 31. Next, at 
S1040. The resulting search range El (the area enclosed by step S1180, half of the eye x-coordinate difference dx 
the dotted line in FIG. 3) from (x3-dx) to (x3+dx) in the («x2~xl), i.e., 
x-axis direction, and 

(*-!)»(*♦!) 



25 dx 

2 * 



dx i x2-xl\ 
*4 I = 4 J' 



is obtained to define the eye searching range in the x-axis 
in the y-axis direction, is thus defined as the searching range 30 direction based on the user-defined center point pi of left 
for mouth m. eye el. One-fourth of the eye x-coordinate difference dx, 

An alternative method of obtaining the mouth m search- i e » 
ing range is shown in FIG. 4. Now referencing the flow chart 
of FIGS. 17 and 18, control within the mouth searching 
range setting unit 31 begin at step S1050. Control thereafter 35 
passes to step S1060 in which the image data and user 

defined point for the mouth and eyes are acquired. Control fe obtained t0 de fi ne me eye searching range in the y-axis 
thereafter passes to step S1070 in which mouth searching direction based on the user-defined center point pi. The 
range setting unit 31 obtains the difference dx between the 4o resulting range £3 (the area enclosed by the dotted line in 
x coordinates of center points pi and p2 of left and right eyes pjQ 3^ f rom 
el and e2 (dx=x2-xl), the difference dyl between the y 

coordinate (-yl)ofthe left eye el and they coordinate (=y3) , dx) ( dx\ 

of the mouth (dyl-yl-y3) and the difference dy2 between [ xl " 7 j to [ xJ + J J 

the y coordinate (=y2) of the right eye e2 and the y 45 
coordinate («y3) of the mouth (dy2«y2-y3), at step S1080. 

Control thereafter pass to step S1090, where the average SI in the x-axis direction, and 
of these difference values 



/ {dx + dyl+dy2)\ 50 

r- — 5 — J 



in the y-axis direction, is thus defined as the searching range 
is calculated to define the mouth searching range in the for left eye el. Processing thereafter tenninates naturally at 
x-axis direction based on the user-defined center point p3 of 55 ste P S119 °- ^ searching range for right eye e2 is similarly 
the mouth, as shown in FIG. 4. One-half of this average si defined > based on user-defined center point p2. 
is then obtained Also, preferably concurrently, eyebrows searching range 

setting unit 33 determines the searching ranges for left and 
. sf v right eyebrows bl and b2. The method applied here is the 

[** c - y J 60 same f° r * 50tn eyebrows, so the following description is 

therefore limited to left eyebrow bl. It should be noted here 

that the eyebrow searching ranges are defined based on the 
to define the mouth searching range in the y-axis direction, user-defined center points for the eyes in the present 
also based on the user-defined center point p3 of the mouth embodiment, because there are no user-defined center points 
SHOO (FIG. 18.). The resulting range E2 (the area enclosed 65 specifically for the eyebrows. 

by the dotted line in FIG. 4) from (x3-sl) to (x3+sl) in the Referring to FIG. 5, the x-axis limits of the eyebrow 
x-axis direction, and searching range are defined as 
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than the left and right end point y-coordinates yll and yl2 

& (i.e. yll>yl3 and yl2>yl3), it is determined that the 

j extracted mouth area represents an open mouth, and the area 

extracted as the mouth area represents only the lower lip 
s area. 

from center point pi to the nose-side of the eye center, and i D this case, control passes to step $1240, in which the 

searching range is reset referenced to the left and right end 

dx point coordinates (xll, yll) and (xl2, yl2). This 

~2 re -searching range is the searching range E5 shown by the 

dotted line in FIG. 6. The size of searching range E5 may be, 
for example, an area equal to half the y-axis height of the 

from center point pi to the outside of the eye center (where complete mouth searching range El shown in FIG. 3, i.e., an 

x-coordinate difference dx of the left and nght eyes el, e2 is area ^ i ong on the x . ax is and 
x2-xl as defined above) based on the user-defined center 

point pi for the left eye. The y-axis limits of the eyebrow ^ 

searching range are denned as is — 

2dx dx 

~5~ 10 10 high on the y-axis, but shall not be so limited, as will become 

apparent to those ordinarily skilled in the art. The 
.20 re-searching region may also be optimally defined. Control 

above the user-defined left-eye center point pi, resulting in thereafter terminates naturally at step S1250. 

searching range E4 for left eyebrow bl as also shown in FIG. of if it ^ determined in step S1230 that the lip 

5. The searching range for nght eyebrow b2 is similarly cusp mid point ^ not lower thaQ the corner points of the 

defined, based on user-defined center point p2. mouth> control instead naturally terminates at step 1250 

Once the user defines the center points for the left and without a need to reset ^ mouth range 

nght eyes and mouth, facial image processing according to 25 ^ mouth area can ^ be re liably extracted 

the present embodiment is thus able to define the searching even ^ images in which the sub j ect > s mouth ^ open> and the 

ranges for the mouth, eyes, and eyebrows based on those facial image processing method of this embodiment can thus 

user-defined points. The method for extracting the areas of also be used ^ smiling or u]kiQg sub j ects . 

the mouth, eyes, and eyebrows within the defined searching ^ eye and eye brow areas are extracted in the same way 

ranges according to the first embodiment is described below. 30 ^ me moutn by eye area extractor 42 and eyebrow area 

To extract the area of the mouth once the searching range extractor 43 according to a particular program confined to 

for the mouth has been set by mouth searching range setting the defined searching ranges. For example, each pixel within 

unit 31, mouth area extractor 41 follows a particular pro- the searching range is scanned using a raster scan technique, 

gram confined to the defined searching range. For example, and the YIQ color system I component of each pixel is 

each pixel within the searching range may be scanned using 35 compared with a predetermined threshold value to detect 

a raster scan technique, and the YIQ color system Q com- those pixels for which the I component exceeds the thresh- 

ponent of each pixel is compared with a predetermined old value. If the pixels for which the I component exceeds 

threshold value to detect those pixels in which the Q the threshold value are concentrated in a single area, and the 

component exceeds the threshold value. If the pixels in area of this area is at least equal to a known size, that area 

which the Q component exceeds the threshold value are 40 is determined to be the eyes (eyebrows). Note that if there 

concentrated in a single region, and the area of this region are plural regions within the searching range in which the I 

is at least equal to a known area (to eliminate scanning noise component exceeds the threshold value, the region having 

and slight facial variations), that region is determined to be the greatest area is determined to be the area of the eyes 

the mouth (or more specifically, the lips). Note that if there (eyebrows). 

are plural regions within the searching range in which the Q 45 Because the searching ranges corresponding to the left 

component exceeds the threshold value, the region having and right eyes, mouth, and eyebrows are set based on the 

the greatest area is determined to be the region of the mouth coordinates of the three points defined by the user as the 

(lips) according to this embodiment. centers of the left and right eyes and mouth, and identifi- 

When the mouth is open and the upper and lower lips are cation of facial features can therefore be restricted to those 

separated, such as when the subject in the image is smiling, 50 user-defined areas, the process executed to extract those 

the area of the lower lip is typically greater than the area of features can be greatly simplified, and feature extraction can 

the upper lip, and may be erroneously defined as the entire be more precise. 

mouth area as shown in FIG. 6. To prevent this, mouth The process for setting the facial contour searching range 

extraction content evaluator 61 executes the following pro- by the facial contour searching range setting unit 34 for 

cedure as detailed in the flowchart of FIG. 20. 55 facial contour extraction is described next, using, by way of 

Controls begins at step S1200, and then progresses to step example, the process for setting the facial contour searching 

1210 in which the left and right corner mouth points as range for extracting the facial contour in the area at the 

extracted by mouth contour point extractor 51 are (xll, yll) height of eyes el and e2. The process described below 

and (xl2, yl2) as shown in FIG. 6. Then, at step S1220, the extracts the facial contour in the area on a line extended 

coordinates (xl3, yl3) of the top edge point or lip cusp p ra 60 horizontally (in the x-axis direction) from the user-defined 

of the region extracted as the mouth are obtained at the center points of the left and right eyes. Note that because the 

midpoint between the left and right end points (xll, yll) and same process is symmetrically applied for the left and right 

(xl2, yl2). The relationship between the position of this top eyes, the description below is limited to that for setting the 

edge midpoint y-coordinate yl3 and the y-coordinates yll searching range for extracting contour point pi correspond- 

and yl2 of the left and right end points (the vertical 65 ing to left eye el. 

relationship between these three points) is then investigated As previously described, the center coordinates of left and 

in step S1230. If the midpoint p m y-coordinate yl3 is less right eyes el, e2 are (xl, yl) and (x2, y2), respectively, and 
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the center coordinates of mouth m are (x3, y3). A searching 

range E6 including the contour point pi is then first set as r (yj - y3) + (>2 - y3)\ 

shown in FIG. 7 based on these coordinates. The limits of { = wI ~ \ J 

range E6 in the x-axis direction are set on a horizontal line 
passing through center point pi of the left eye and contour 5 

point pll (the coordinates of which are still unknown). Note is obtained, as shown in step S1320. Then, in step S1340, the 
that the y-axis dimension of the range is not specifically average wl is multiplied by eleven-tenths (^10) to obtain the 
defined, and an appropriate range is therefore set on the distance 
y-axis relative to the horizontal line through point pi. 

Specifically, referring to the flowchart of FIG. 21, control 
passes to step S1300 and S1310, in which the image and user 
defined points are acquired in a manner discussed herein- 
above. Then control passes to step S1320, in which the . , ^ ^ 
average of the y-axis coordinates of left and right eyes el, of the outside edge of searching range E7 from center point 
e2 and mouth m P 3 °f me moum » and multiplied by three-fifths to obtain the 

15 distance 



10 



(yl-y3) + (y2-y3)\ 



Hi) 



is calculated. Control then passes to step S1330, in part, 20 

where the average wl is multiplied by three-fourths to of the inside edge of searching range E7 from center point 
obtain the distance P 3 °f tne mouth. Again, the same process is symmetrically 

applied to obtain the x-axis limits of the searching range for 
✓ 3\ the facial contour on the other side of the face. 

|>v/x-J 25 Facial contour extractor 44 is then able to extract the 

facial contour within this range by applying an edge detec- 
tion process, e.g., scanning the pixels in this range to extract 
of the outside edge of searching range E6 from center point those pixels at which there ^ a significant luminance dif- 
pl of the left eye, and multiplied by one-third to obtain the f er ence. It is to be noted that an appropriate height limit is 
distance 3Q defined for range E7 in the y-axis direction referenced 

to the horizontal line through contour point pl2 as shown in 
FIG. 7. This is to obtain the average contour line when 
irregularities are created in the contour line due to noise or 
other factors in the edge detection process, 
of the inside edge of searching range E6 from center point 35 A similar process is next executed to set the facial contour 
pi of the left eye. The same process is symmetrically applied searching range for extracting the facial contour in the area 
to obtain the x-axis limits of the searching range for the of the chin as described below. In this case contour pomt pl3 
facial contour on the other side of the face. Control then on a line extended vertically (in the y-axis direction) from 
naturally terminates at step S1360. the mouth m center (user-defined point p3) is extracted. 

Facial contour extractor 44 is then able to extract the 40 As described above, searching range E8 including the 
facial contour within this range by applying an edge detec- contour P omt P 13 » set as shown in FIG. 7 based on the 
tion process, e.g., scanning the pixels in this range to extract previously obtained center coordinates of left and nght eyes 
those pixels there is a significant luminance difference. It is el > e2 ' X 1 ) and i* 2 * V 2 )* respectively, and mouth m, (x3, 
to be noted that an appropriate height limit is also defined for V 3 )- ^ b ' mits of range E8 in the y-axis direction are set on 
range E6 in the y-axis direction referenced to the horizontal 45 a vertical line P^g trough P oint P 3 of the mouth 
line through contour point pi as shown in FIG. 7. This is to and oontou '' P oiat P 13 Note that x " axis dimension of the 
obtain the average contour line when irregularities are range is not specifically defined, and an appropriate range is 
created in the contour line due to noise or other factors in the the . refore ^ on tDe x * axis relative t0 the vertical line 
edge detection process. point pl3. 

A similar process is executed preferably in parallel to set 50 Specifically still referring to the flow chart of FIG. 21, the 
the facial contour searching range for extracting the facial avera 8 e of the y-*™ coordinates of left and nght eyes el, 
contour in the area at the height of the mouth as described e2 311(1 mouth m 
below. In this case contour point pl2 on a line extended 

horizontally (in the x-axis direction) from the mouth m ( _ wJ = iy* -y 3 ) + (y 2 ~y 3 A 

center (user-defined point p3) is extracted. 55 I 2 J 

As described above, searching range E7 including the 
contour point pl2 is set as shown in FIG 7 based on the fe obtained M shown m sl320 in mis ^ 

previously obtained center coordinates of left and nght eyes braQches tQ S1350 m which ^ a e wl fa 

el, e2, (xl, yl) and (x2, y2), respectively, and mouth m, (x3, multiplied 5 thre e-fourths to obtain the distance 
y3). The limits of range E7 in the x-axis direction are set on 60 
a horizontal line passing through center point p3 of the 

mouth and contour point pl2. Note that the y-axis dimension hw x - j 

of the range is not specifically defined, and an appropriate 
range is therefore set on the y-axis relative to the horizontal 

line through points p3 and pl2. 65 of the outside (bottom) edge of searching range E8 from 

Specifically referring to FIG. 21 the average of the y-axis center point p3 of the mouth, and multiplied by one-half to 
coordinates of left and right eyes el, e2 and mouth m obtain the distance 
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of the inside (top) edge of searching range E8 from center 
point p3 of the mouth. 

Facial contour extractor 44 is then able to extract the 
facial contour within this range by applying an edge detec- 
tion process, e.g., scanning the pixels in this range to extract 
those pixels at which there is a significant luminance dif- 
ference. It is to be noted that an appropriate horizontal limit 
is also defined for range E8 in the y-axis direction referenced 
to the vertical line through contour point pl3 as shown in 
FIG. 7. This is to obtain the average contour line when 
irregularities are created in the contour line due to noise or 
other factors in the edge detection process. 

By thus separately defining searching ranges for and 
extracting the mouth, eyes, eyebrows, and parts of the facial 
contour, the coordinate data for the contour points represen- 
tative of the corresponding facial features can be extracted 
by mouth contour point extractor 51, eye contour point 
extractor 52, eyebrow contour point extractor 53, and facial 
contour point extractor 54. The extracted contours are then 
verified by the corresponding mouth extraction content 
evaluator 61, eye extraction content evaluator 62, eyebrow 
extraction content evaluator 63, and facial contour extrac- 
tion content evaluator 64. If no errors are detected in the 
extracted contents, the coordinate data for the contour points 
extracted by the contour point extractors 51-54 is output. 

The verification process executed by the extraction con- 
tent evaluators 61-64 determines whether the areas 
extracted by the mouth, eyes, eyebrows, and facial contour 
extraction processes are appropriate. An example of these 
processes for determining whether the eyes and mouth were 
correctly extracted is described below. 

Whether the defined searching ranges are usable (valid) 
can be determined by determining whether the computed 
ranges contain the corresponding user-defined points. For 
example, if the user-defined point for extracting an eye is not 
the eye center but a point closer to the eyebrow, the set 
searching range may contain the eyebrow. As a result, pixel 
scanning may result in the eyebrow being falsely extracted. 
If the eyebrow is wrongly extracted when the eye should be 
extracted as in this example, it is possible to determine 
whether the extracted object is correct by checking whether 
the point defined by the user as the eye center is contained 
within the area of the eyebrow. Because it is the eye and 
eyebrow that are checked in this case, the evaluation is based 
on the relationship between the y-axis coordinates of the eye 
and eyebrow, specifically by determining whether the user- 
defined y-coordinate of the eye is within the maximum and 
minimum y-axis coordinates of the extracted area. If the 
user-defined point is within the extracted area, the extracted 
area is correct; if not, the extracted area is wrong. If the 
extracted area is determined to be wrong, a request to repeat 
the process is output. 

In the case of the mouth, it is both determined whether 
only the lower Hp was extracted, as described above, and 
whether the user-defined point (mouth center) is within the 
extracted area. The specific process executed is substantially 
identical to that used to verify the eyes as described above. 

In the case of the eyebrows, however, there is no user- 
defined eyebrow center point (coordinates), and it is there- 
fore not possible to verify the extracted eyebrow area by 
checking whether a user-defined point is contained therein. 
The eyebrow areas are therefore verified referenced to the 
corresponding user-defined eye centers, specifically by com- 
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paring the user-defined eye coordinates with the coordinates 
of the area extracted as the eyebrow. 

This is described with reference to FIG. 8 in the flow chart 
of FIG. 22. As shown in FIG. 22 control begins at step S1400 

5 and progresses to step S1410 in which the left and right 
corner points of the subject eyebrow contain within the 
image are acquired, e.g., (x21, y21) and x22, y22 respec- 
tively as shown in FIG. 8A. Next control passes to step 
S1420 in which determination is made where the eye 

10 adjacent to the subject eyebrows horizontally between the 
acquired eyebrow corner points. If so, control passes to step 
S1430. At step S1430, a determination is made whether the 
eyebrow corner points are vertically displaced above the 
adjacent eye. If so, control terminates naturally at step 

15 S1450, as the eyebrows have been correctly abstracted. 
If however in step S1430 or in step S1420, either deter- 
mination is made that the eyebrow corner points are mis- 
placed with respect to the predefined subject eye point, 
control instead passes to step S1440. At step S1440, a 

20 request is made to reprocess the area as evidently the 
eyebrow corner point fall outside the acceptable range with 
respect to the predefined adjacent eye point. Control there- 
after terminates naturally at step S1450. This evaluation 
criterion makes it possible to determine that area h (see FIG. 

25 8B), resulting from hair hanging down at the left side of the 
face being extracted as the eyebrow area, is not a valid 
eyebrow area because the statement x31<xl<x32, where 
x31 and x32 are respectively the left and right side x-axis 
coordinates of area b, is not true. 

30 It is to be noted that all of the coefficient values, e.g., Vi, 
2 /s t and Vio, used in the above embodiment for setting the 
various searching ranges are sample values only, and various 
other values may be alternatively used as will be appreciated 
by those ordinarily skilled in the art. 

35 Second Embodiment 

The first embodiment of the present invention was 
described above as a methodology for setting the searching 
ranges for the mouth, eyes, eyebrows, and other facial 
features, extracting those facial features within the set 

40 searching ranges, and then outputting the resulting coordi- 
nate data. A method for reliably extracting the coordinate 
data of the representative contour points of the eyes, mouth, 
and other facial feature contours without being affected by 
noise is described according to the second embodiment of 

45 the present invention hereinbelow. It is to be noted that 
extracting the areas of the eyes, mouth, and other feature 
contours shall not be limited to the first embodiment above 
whereby a searching range is first set and the facial features 
are then extracted within those areas. 

50 FIG. 9 is a block diagram used to describe this second 
embodiment. As shown in FIG. 9, this method comprises an 
image memory 2, feature unit extractor 4, contour point 
extractor 5, and extracted content evaluator 6. Feature unit 
extractor 4 of this embodiment is equivalent to the mouth 

55 area extractor 41, eye area extractor 42, eyebrow area 
extractor 43, or facial contour extractor 44 of the first 
embodiment, and may similarly be means for extracting 
feature units based on the image data within defined search- 
ing range, but shall not be so limited. 

60 Contour point extractor 5 similarly corresponds to mouth 
contour point extractor 51, eye contour point extractor 52, 
eyebrow contour point extractor 53, or facial contour point 
extractor 54, and extracted content evaluator 6 to mouth 
extraction content evaluator 61, eye extraction content 

65 evaluator 62, eyebrow extraction content evaluator 63, or 
facial contour extraction content evaluator 64 of the first 
embodiment as shown in FIG. 1. 
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With the second embodiment thus comprised, it is nec- 
essary to extract the coordinate values of contour points p21 
. . . p36 forming the contour of the eye, and point p37 
representing the pupil, as shown in FIG. 10 based on the 
area, for the one eye in this example, extracted by feature 
unit extractor 4. To extract the coordinates of the points p21 
. . . p36 forming the eye contour, the most common method 
as shown in FIG. 23 (steps S1500-S1540) is to first extract 
the coordinates of the two end points p21 and p29 in the 
x-axis direction (specifically, the right and left end points at 
the longest part in the x-axis direction of the area extracted 
as the eye), and then extract the coordinates of the two top 
and bottom points p25 and p33 in the y-axis direction 
(specifically, the top and bottom points with the greatest 
distance therebetween in the y-axis direction of the area 
extracted as the eye). The coordinates of the two top and 
bottom points p25 and p33 are extracted by this second 
embodiment of the present invention by first defining a 
searching range (dotted line area E9 as shown in FIG. 10) 
including the midpoint between the right and left end points 
p21 and p29 (Step S1520), and then searching for the top and 
bottom points p25 and p33 only within searching range E9. 

The size of searching range E9 is determined by obtaining 
the range 

dx 

(where dx=x2-xl as described in the first embodiment) 
above and below the user-defined eye center coordinates in 
the y-axis direction (S1530), and one-tenth of the length 
between right and left eye contour end points p21 and p29 
on either side of the eye center coordinates in the x-axis 
direction. It is to be noted, however, that the present inven- 
tion shall not be limited to this range definition, and other 
searching range sizes can also be used in particular appli- 
cations. 

Why the searching range for obtaining the top and bottom 
end points of the eye contour is based on the midpoint 
between the right and left end points of the eye contour is 
described below. 

Specifically, the overall shape of the eyes and mouth is 
generally oval. The top and bottom points p25 and p33 in the 
y-axis direction (specifically, the top and bottom points with 
the greatest distance therebetween in the y-axis direction) 
are therefore normally located in the middle between the 
right and left ends of the eye. However, if there are so-called 
"whiskers" caused by noise in the area above or below the 
right or left end points p21, p29 of the eye, those noise points 
may be erroneously processed as the top and bottom end 
points. If the coordinates of these noise points are extracted 
as the top and bottom end points, and the image is then 
processed using this coordinate data, unnaturally-shaped, 
undesirable eye contours may result. To prevent this, the 
searching range for the top and bottom points p25 and p33 
is restricted to an area in the middle of the eye contour. Note 
that this principle is not limited to the eyes, and is similarly 
applied when extracting the mouth coordinate data. 

In the case of the mouth, a searching range (dotted area 
E10 in FIG. 11) is similarly defined to locate the top and 
bottom end points. Referring to FIG. 11, points p41 . . . p56 
define the outside contour of the mouth (lips). The size of 
searching range E10 is determined by obtaining the range 
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dx 

2 

5 (where dx-x2-xl as described in the first embodiment) 
above and below the user-defined mouth center coordinates 
in the y-axis direction, and one-tenth of the length between 
right and left mouth contour end points p41 and p49 on 
either side of the mouth center coordinates in the x-axis 

10 direction. It is to be noted, however, that the present inven- 
tion shall not be limited to this range definition, and other 
searching range sizes can also be used. 

Because the area of the mouth is relatively large compared 
with the area of the eyes, limiting the size of the searching 

15 range in the y-axis direction as shown in FIG. 11 greatly 
reduces the processing time required to find the top and 
bottom end points. Furthermore, in images in which the 
subject's mouth is open and the lips are extended to the 
sides, symmetrical contour points on the outside contour of 

20 the lips are substantially level (points p43 and p47, and 
points p55 and p51, for example, are essentially level in FIG. 
11), the area of the searching range for the top and bottom 
end points is larger, it can be difficult to accurately identify 
the desired end points, and false contour points caused by 

25 noise in the horizontal areas may be wrongly processed as 
the end points. A searching range as shown by the dotted 
area E10 in FIG. 11 is therefore set, and the top and bottom 
end points are obtained only within this searching range. 
After the right and left end point coordinates and the top 

30 and bottom end point coordinates are obtained, the coordi- 
nates of the contour points between the end points are 
obtained. In the case of the mouth, for example, the coor- 
dinates of point p43 and the symmetrical point p55 (i.e., top 
and bottom contour points having the same x-axis 

35 coordinate), and point p47 and the symmetrical point p51 
(again, the top and bottom contour points having the same 
x-axis coordinate), are obtained. 

If the eye is slightly slanted as shown in FIG. 12, the 
coordinates of the points between the right and left end 

40 points and top and bottom end points are obtained as 
described below based on the coordinates of the previously- 
obtained right and left end points p21, p29 and top and 
bottom end points p25, p33. Note that the coordinates of 
point p23 and the corresponding point p35 (the x-axis 

45 coordinates of which are different in this case because the 
eye is slanted), and point p27 and the corresponding point 
p31 (also having different x-axis coordinates), are used as 
the midpoints to be obtained below. 
For example, referring to FIG. 12, the x-axis coordinate of 

50 point p23 is the average of the x-axis coordinates of points 
p21 and p25, and the y-axis coordinate is the maximum 
y -coordinate candidate in the area of that x-axis coordinate. 
The x-axis coordinate of point p35 is likewise the average of 
the x-axis coordinates of points p21 and p33, and the y-axis 

55 coordinate is the maximum y-coordinate candidate in the 
area of the selected x-axis coordinate. The coordinate data 
for points p27 and p33 is obtained in the same manner. 

Once the coordinate data of each of the contour points is 
thus obtained, extracted content evaluator 6 determines 

60 whether there are any errors in the extracted content. If not, 
the coordinate data thus obtained is output. 

An example of the evaluation process executed by 
extracted content evaluator 6 is described below. For 
example, the lighting may create shadows causing contour 

65 points within shaded area Al in FIG. 13 to not be extracted, 
resulting in a slanted mouth image as shown in FIG. 13. In 
the case shown in FIG. 13, the subject was illuminated by 
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lighting from the right side, resulting in shadow Al. While facial contour. It is therefore extremely simple to specify 

the correct shape of the mouth should normally include the these points. Furthermore, because the mouth, right and left 

shaded area Al, the color data of the pixels in this area is eyes, eyebrows, and the facial contour are extracted using 

changed by the lighting, and the shape of the extracted searching ranges defined according to the coordinates of 

mouth profile slopes down to the left as indicated by line LI. 5 these pre-defined points, most of the primary facial features 

Contours extracted in this manner can be determined to be can be reliably and simply extracted, and the searching 

correct or incorrect by checking the slope of the extracted ranges can be dynamically set for faces of different sizes, 

mouth contour. Specifically, if the slope of line LI connect- Also, the searching range for the mouth area may be set 

ing left and right end points ml and m2 of extracted mouth in the x-axis and y-axis directions to include the mouth area 

m is greater than a predetermined angle relative to line L2 10 referenced to the point specified as the mouth based on the 

connecting the center points of the left and right eyes el and difference between the x-coordinate data of the points speci- 

e2, the extracted mouth contour is determined to be wrong fied as the right and left eyes, where the x-coordinate axis 

and re-processing is requested. The angle appropriate as this represents the side -to-side direction of the facial image and 

predetermined angle has been demonstrated through tests to the y-coordinate axis represents the vertical direction of the 

be ±7°. is facial image. It is therefore possible to set an optimal 

When obtaining the coordinates of the contour points searching range for extracting the mouth area, and the mouth 

forming the facial contour, any particular extracted contour area can be simply and reliably extracted, 

point may be positioned closer to the inside of the face (the Also, the searching range for the mouth area may be set 

nose side of the facial contour) relative to the two adjacent in the x-axis and y-axis directions to include the mouth area 

contour points. This is shown, for example, in FIG. 14 where 20 referenced to the point specified as the mouth based on the 

contour point p73 is positioned noticeably to the inside of difference between the x-coordinate data of the points speci- 

the facial contour. This may result when there is a beard or fied as the right and left eyes, and the average of the 

conspicuous skin wrinkle in that area, thus causing a lumi- differences between the y-axis coordinate of the point speci- 

nance value exceeding the threshold value to be detected and fied as the mouth and the y-axis coordinates of the points 

returned as a false contour point. When this happens, the 25 specified as the right and left eyes. It is therefore possible to 

facial contour becomes significantly concave in that area, set an even more optimal searching range for extracting the 

and an unnatural facial image results. mouth area, and the mouth area can be simply and reliably 

To resolve this problem, the average value of the coordi- extracted, 
nates of the two adjacent contour points is obtained, and it Moreover, the searching ranges for the eyes may be set in 
is detected whether the coordinates of the contour point 30 the x-axis and y-axis directions to include the eyes refer- 
between these two contour points are positioned to the inside enced to the points specified as the right and left eyes based 
of the face from this average coordinate point. If the middle on the difference between the x-axis coordinate data of the 
contour point is to the inside, the average of the coordinates points specified therefor. It is therefore possible to set an 
is substituted for the coordinates of that inside contour point. optimal searching range for extracting the eye areas, and the 
This is described with reference to FIG. 14. 35 eye areas can be simply and reliably extracted. 

As shown in FIG. 14, contour point p73 can be deter- Further, as discussed hereinabove, the searching ranges 

mined to be inside of average coordinate point p73', which for the eyebrows are set in the x-axis and y-axis directions 

is the average of points p72 and p74, by comparing the to include the eyebrows referenced to the points specified as 

respective coordinate data. As a result, the coordinates for the right and left eyes and based on the difference between 

average coordinate point p73' are substituted for the coor- 40 the x-axis coordinate data of the points specified as the right 

dinates of contour point p73. The facial contour is therefore and left eyes. It is therefore possible to set an optimal 

approximated by the dot-dash line curve linking contour searching range for extracting the eyebrow areas, and the 

points p71, p72, p73\ and p74, and the resulting facial eyebrow areas can be simply and reliably extracted, 

contour more closely approximates the natural facial image. Further still, the searching ranges for the facial contours 

By means of a specific point being pre-defined in the 45 in the areas at the eye height in the y-axis direction may be 

facial image data for each of plural facial features to be set in the x-axis and y-axis directions to include said facial 

extracted, facial image processing according to the present contours referenced to the points specified as the right and 

invention as described hereinabove sets, for each facial left eyes and based on the average of the differences between 

feature to be extracted, a searching range for extracting said the y-axis coordinate of the point specified as the mouth and 

facial features based on the coordinate data of said specific 50 the y-axis coordinates of the points specified as the right and 

points and referenced to the specific point specified for each left eyes. It is therefore possible to set optimal searching 

facial feature, and extracts the area of each of the facial ranges for extracting the facial contours in the areas at eye 

features within the corresponding searching range. As a height, and the facial contour areas can be simply and 

result, processing is greatly simplified when compared with reliably extracted. 

methods extracting facial features from the entire facial 55 Also, the searching range for the facial contour in the area 

image, and processing time can therefore be greatly reduced, at the mouth height in the y-axis direction may be set in the 

because the facial features are extracted from a limited x-axis and y-axis directions to include said facial contour 

searching range. Furthermore, because the respective facial referenced to the point specified as the mouth and based on 

features are extracted from within defined image areas, false the average of the differences between the y-axis coordinate 

extraction of other facial features in place of the desired 60 of me point specified as the mouth and the y-axis coordinates 

facial features rarely occurs, and the facial feature the user of the points specified as the right and left eyes. It is 

wishes to extract can be reliably extracted. therefore possible to set optimal searching ranges for 

Further, the facial features for which said specific points extracting the facial contours in the areas at the mouth 

are set are the right and left eyes and the mouth, and the height, and the facial contour areas can be simply and 

facial features for which the area is extracted after setting the 65 reliably extracted. 

searching ranges based on the coordinate data for said three Likewise, the searching range for the facial contour in the 

points are the mouth, right and left eyes, eyebrows, and the area of the mouth in the x-axis direction may be set in the 
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x-axis and y-axis directions to include said facial contour 
referenced to the point specified as the mouth based on the 
average of the differences between the y-axis coordinate of 
the point specified as the mouth and the y-axis coordinates 
of the points specified as the right and left eyes. It is 5 
therefore possible to set an optimal searching range for 
extracting the facial contour in the chin area, and the facial 
contour can be simply and reliably extracted. 

According to a further aspect of the invention, a specific 
point is defined in the facial image data for each of plural 10 
facial features to be extracted; a searching range for extract- 
ing the areas of said facial features is set for each facial 
feature to be extracted based on the coordinate data of said 
specific points and referenced to the specific point specified 
for each facial feature; the areas of said facial features are is 
extracted within the set searching ranges; the coordinate data 
of the contour points forming each facial feature contour is 
extracted from the facial feature areas; and the relative 
positions of the extracted feature images are evaluated based 
on the coordinate values of specific points in the facial 20 
feature areas to determine whether the facial features were 
correctly extracted. It is therefore possible to check whether 
facial feature extraction was correctly executed and to 
request reprocessing if there is an extraction error, and it is 
therefore possible to eliminate extraction errors. 25 

Also, the evaluation process for evaluating correct extrac- 
tion of the mouth area extracted in the searching range set 
therefor may compare the y-axis coordinates of the right and 
left end points in the x-axis direction of the extracted mouth 
area with the y-axis coordinate of the midpoint in the x-axis 30 
direction at the top edge of the extracted mouth area. In this 
case, the y-axis coordinate of the midpoint in the x-axis 
direction is smaller than the y-axis coordinate of either right 
or left end point in the x-axis direction of said mouth area, 
the extracted mouth area is determined to represent an open 35 
mouth, the extracted mouth image is determined to be the 
lower lip only, and a searching range for extracting the upper 
lip is therefore set. Extraction of only the lower lip area can 
therefore be prevented in facial images in which the subject 
is smiling, for example, and the entire mouth area can be 40 
extracted when the mouth is open in the image. 

Further, the evaluation process for determining correct 
extraction of the eyes or mouth may determine whether the 
eyes or mouth were correctly extracted by determining 
whether the respective points specified by the user as the 45 
center of the eyes or mouth are contained within the 
extracted areas. It is therefore possible to easily and reliably 
determine whether the eyes or mouth were correctly 
extracted. 

Further still, the evaluation process for determining cor- 50 
rect extraction of the eyebrows determines whether the 
eyebrow areas were correctly extracted by evaluating the 
positions of the coordinates of the right and left end points 
of the extracted eyebrow area relative to the coordinates of 
the point specified as the corresponding eye. It is therefore 55 
possible to easily and reliably determine whether the eye- 
brow areas were correctly extracted. 

According to the second embodiment of the present 
invention, the contour points forming the contours of the 
eyes and mouth in the corresponding facial feature areas are 60 
extracted after extracting the area of each facial feature, the 
coordinates of the right and left end points in the x-axis 
direction of said facial feature areas are then obtained from 
said contour points; a searching range with a width in the 
x-axis direction equal to a known Vn fraction of the distance 65 
between the right and left end points of the respective facial 
feature area is set in each area near the midpoint between 
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said right and left end points; and the coordinates of the top 
and bottom end points in the y-axis direction of the eye or 
mouth facial feature areas are then obtained within said 
searching range. As a result, the contour points at the highest 
and lowest parts of said contour lines can be reliably 
obtained without being affected by noise. 

Further, according to this embodiment, the coordinates of 
the contour points between the obtained four right, left, top, 
and bottom end points are obtained with the x-axis coordi- 
nate thereof calculated as the average of the x-axis coordi- 
nates of the contour points on either side of the contour point 
being calculated. It is therefore possible to easily and 
reliably extract the intermediate contour points between the 
end points, particularly in facial images where the eye 
profiles are slanted or drooping. 

Also, the relative positions of the extracted facial feature 
areas are evaluated based on the coordinate values of a 
specific point in each facial feature area to determine 
whether the facial feature areas were correctly extracted. It 
is therefore possible to check whether facial feature extrac- 
tion was correctly executed and to request reprocessing if 
there is an extraction error, and it is therefore possible to 
eliminate extraction errors. 

For the second embodiment disclosed hereinabove, this 
evaluation process for evaluating correct extraction of the 
mouth area may determine whether the extracted area is 
correct by determining whether a line connecting the right 
and left end points in the x-axis direction of the mouth area 
is sloped greater than a predetermined angle relative to a line 
through a specific point in each of the right and left eyes. It 
is therefore possible to detect when the extracted image area 
is unnaturally curved as a result lighting-induced shadowing 
causing part of the mouth area image to be lost. In addition, 
evaluation errors can be prevented when extracting facial 
features when the entire facial image is on an angle because 
the angle of the mouth is also referenced to a known line 
passing through the eyes. 

Finally, the evaluation process for evaluating correct 
extraction of the facial contour area may process the plural 
contour points forming the facial contour in units of three 
adjacent contour points, compares the coordinates of the 
middle of each three contour points with the average of the 
coordinates of the two adjacent contour points, and substi- 
tutes the average of the coordinates of the two adjacent 
contour points for the coordinates of the middle contour 
point if the position of the middle contour point is deter- 
mined to be to the inside of the face relative to the adjacent 
contour points. It is therefore possible to determine when the 
facial contour is unnaturally concave due to a beard or 
pronounced skin wrinkle in the facial image. Correction of 
such unnatural concave facial curves can therefore be easily 
accomplished, and a more natural facial contour can be 
obtained. 

While the invention has been described in conjunction 
with several specific embodiments, it is evident to those 
skilled in the art that many further alternatives, modifica- 
tions and variations will be apparent in light of the foregoing 
description. Thus, the invention described herein is intended 
to embrace all such alternatives, modifications, applications 
and variations as may fall within the spirit and scope of the 
appended claims. 

What is claimed is: 

1. A facial image processing method for extracting facial 
features and outputting the corresponding coordinate data 
based on facial image data, wherein: 

a specific point is defined in the facial image data for each 
of the right and left eyes and the mouth; 
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a searching range is set for each facial feature to be 
extracted, including the right and left eyes and the 
mouth, based on the coordinate data of at least one of 
said specific points and referenced to the specific point 
specified for that facial feature for which the searching 
range is being set, 

where the x-coordinate axis represents the side-to-side 
direction of the facial image and the y-coordinate 
axis represents the vertical direction of the facial 
image; 

an area of each of the right and left eyes and the mouth is 
extracted within the corresponding searching range; 
and 

it is determined whether the extracted mouth area is 
correct by determining whether a line connecting right 
and left end points in the x-axis direction of the mouth 
area is greater than a predetermined angle relative to a 
line through a particular point in each of the right and 
left eyes. 

2. A facial image processing method according to claim 1, 
wherein 

a searching range is set for each of plurality of additional 
facial features including the eyebrows, a facial contour 
to the outside of the right eye and a facial contour to the 
outside of the left eye, wherein the searching range for 
each of the facial contours outside of the eyes is set in 
the x-axis and y-axis directions to include that facial 
contour referenced to the point specified as the corre- 
sponding eye based on the average of the differences 
between the y-axis coordinate of the point specified as 
the mouth and the y-axis coordinates of the points 
specified as the right and left eyes; and an area of each 
of the additional facial features for which a searching 
range is set is extracted within the corresponding 
searching range. 

3. A facial image processing method according to claim 2, 
wherein the searching range for the mouth area is set in the 
x-axis and y-axis directions to include the mouth area 
referenced to the point specified as the mouth based on 

the difference between the x-coordinate data of the points 
specified as the right and left eyes. 

4. A facial image processing method according to claim 2, 
wherein the searching range for the mouth area is set in the 
x-axis and y-axis directions to include the mouth area 
referenced to the point specified as the mouth based on; 

the difference between the x-coordinate data of the points 
specified as the right and left eyes, and 

the average of the differences between the y-axis coordi- 
nate of the point specified as the mouth and the y-axis 
coordinates of the points specified as the right and left 
eyes. 

5. A facial image processing method according to claim 2, 
wherein the searching ranges for the eyes are set in the 
x-axis and y-axis directions to include the eyes referenced to 
the points specified as the right and left eyes based on 

the difference between the x-axis coordinate data of the 
points specified as the right and left eyes. 

6. A facial image processing method according to claim 2, 
wherein the searching ranges for the eyebrows are set in the 
x-axis and y-axis directions to include the eyebrows refer- 
enced to the points specified as the right and left eyes based 
on 

the difference between the x-axis coordinate data of the 
points specified as the right and left eyes. 

7. A facial image processing method according to claim 1, 
wherein a searching range for facial contour in an area at the 
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mouth height in the y-axis direction is set in the x-axis and 
y-axis directions to include said facial contour in the area at 
the mouth height referenced to the point specified as the 
mouth based on 
the average of the differences between the y-axis coordi- 
nate of the point specified as the mouth and the y-axis 
coordinates of the points specified as the right and left 
eyes. 

8. A facial image processing method according to claim 2, 
wherein a searching range for facial contour in an area on at 
least one side of the mouth in the x-axis direction is set in 
the x-axis and y-axis directions to include said facial contour 
in the area on at least one side of the mouth referenced to the 
point specified as the mouth based on 

the average of the differences between the y-axis coordi- 
nate of the point specified as the mouth and the y-axis 
coordinates of the points specified as the right and left 
eyes. 

9. A facial image processing method for extracting facial 
features and outputting the corresponding coordinate data 
based on facial image data, wherein: 

a specific point is defined in the facial image data for each 
of the right and left eyes and the mouth; 

a searching range for extracting the areas of said facial 
features is set for each facial feature to be extracted, 
including the right and left eyes, the mouth and at least 
one facial contour, based on the coordinate data of at 
least one of said specific points and referenced to the 
specific point specified for that facial feature for which 
the searching range is being set, 

where the x-coordinate axis represents the side-to-side 
direction of the facial image and the y-coordinate axis 
represents the vertical direction of the facial image; 

the areas of said facial features are extracted within the set 
searching ranges; and 

the coordinate data of contour points forming each facial 
feature contour is extracted from the facial feature 
areas; and 

the relative positions of the extracted feature images are 
evaluated based on the coordinate values of specific 
points in the facial feature areas to determine whether 
the facial features were correctly extracted, wherein the 
evaluation process for determining correct extraction of 
the mouth area includes determining whether a line 
connecting right and left end points in the x-axis 
direction of the mouth area is greater than a predeter- 
mined angle relative to a line through a particular point 
in each of the right and left eyes. 

10. A facial image processing method for extracting facial 
features and outputting the corresponding coordinate data 
based on facial image data, wherein: 

a specific point is defined in the facial image data for each 
of plural facial features to be extracted; 

a searching range for extracting the areas of said facial 
features is set for each facial feature to be extracted 
based on the coordinate data of said specific points and 
referenced to the specific point specified for each facial 
feature; 

the areas of said facial features are extracted within the set 
searching ranges; 

the coordinate data of the contour points forming each 
facial feature contour is extracted from the facial fea- 
ture areas; and 

the relative positions of the extracted feature images are 
evaluated based on the coordinate values of specific 
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points in the facial feature areas to determine whether 
the facial features were correctly extracted; 
wherein the evaluation process for determining correct 
extraction of the mouth area extracted in the searching 
range set for extracting the mouth area: 
compares the y-axis coordinates of the right and left 
end points in the x-axis direction of the extracted 
mouth area with the y-axis coordinate of the mid- 
point in the x-axis direction at the top edge of the 
extracted mouth area 

where the x-coordinate axis represents the side-to- 
side direction of the facial image and the 
y-coordinate axis represents the vertical direction 
of the facial image; and 
determines the extracted mouth area to represent an 
open mouth when the y-axis coordinate of the mid- 
point in the x-axis direction is smaller than the y-axis 
coordinate of either right or left end point in the 
x-axis direction of said mouth area, 
determines the extracted mouth to be the lower lip 
only, and 

sets a searching range for extracting the upper lip. 

11. A facial image processing method according to claim 
9, wherein the evaluation process for determining correct 
extraction of the eyes or mouth determines whether the eyes 
or mouth were correctly extracted by 

determining whether the respective points specified for 
the eyes or mouth are contained within the extracted 
areas. 

12. A facial image processing method according to claim 
9, wherein the plurality of facial features includes the 
eyebrows, wherein a searching range is set for the eyebrows, 
and wherein the evaluation process for determining correct 
extraction of the eyebrows determines whether the eyebrow 
areas were correctly extracted by 

evaluating the positions of the coordinates of right and left 
end points of the extracted eyebrow area relative to the 
coordinates of the points specified as the eyes. 

13. A facial image processing method for extracting facial 
features and outputting the corresponding coordinate data 
based on facial image data, wherein 

the area of each facial feature is extracted using a coor- 
dinate system wherein the x-coordinate axis represents 
the side-to-side direction of the facial image and the 
y-coordinate axis represents the vertical direction of the 
facial image, 

the contour points forming the contours of the eyes and 
mouth in the corresponding facial feature areas 
extracted by the facial feature extracting means are 
obtained, 

the coordinates of the right and left end points in the 
x-axis direction of said facial feature areas are then 
obtained from said contour points, 

a searching range with a width in the x-axis direction 
equal to a known Vu fraction of the distance between the 
right and left end points of the respective facial feature 
area is set in each area near the midpoint between said 
right and left end points, and 

the coordinates of the top and bottom end points in the 
y-axis direction of the eye or mouth facial feature areas 
are obtained within said searching range. 

14. A facial image processing method according to claim 
13, wherein: 

the coordinates of the contour points between the obtained 
four right, left, top, and bottom end points are obtained 
with the x-axis coordinate thereof calculated as the 
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average of the x-axis coordinates of the contour points 
on either side of the contour point being calculated. 

15. A facial image processing method for extracting facial 
features and outputting the corresponding coordinate data 
based on facial image data, wherein: 

after extracting the areas of the facial features, the relative 
positions of the extracted areas are evaluated based on 
the coordinate values of a specific point in each facial 
feature area to determine whether the facial feature 
areas were correctly extracted; and 

when the extracted area is the mouth area, it is determined 
whether the extracted area is correct by determining 
whether a line connecting the right and left end points 
in the x-axis direction of the mouth area is sloped 
greater than a predetermined angle relative to a line 
through a specific point in each of the right and left 
eyes, 

where the x-coordinate axis represents the side-to side 
direction of the facial image and the y-coordinate 
axis represents the vertical direction of the facial 
image. 

16. A facial image processing method for extracting facial 
features and outputting the corresponding coordinate data 
based on facial image data, wherein: 

after extracting the areas of the facial features, the relative 
positions of the extracted areas are evaluated based on 
the coordinate values of a specific point in each facial 
feature area to determine whether the facial feature 
areas were correctly extracted; and 
when the extracted area is a facial contour area, the plural 
contour points forming the facial contour are processed 
in units of three adjacent contour points, 
the coordinates of the middle of each three contour points 
are compared with the average of the coordinates of the 
two adjacent contour points, and 
the average of the coordinates of the middle contour point 
if the position of the substituted for the coordinates of 
the middle contour point if the position of the middle 
contour point is determined to be to the inside of the 
face relative to the adjacent contour points, 
where the x-coordinate axis represents the side-to-side 
direction of the facial image and the y-coordinate 
axis represents the vertical direction of the facial 
image. 

17. A facial image processing apparatus for extracting 
facial features and outputting the corresponding coordinate 
data based on facial image data comprises: 

facial feature searching range setting means for setting, 
for each facial feature to be extracted, a searching range 
based on the coordinate data of specific points and 
referenced to a particular one of the specific points 
specified for that facial feature, 
said specific points being pre-defined in the facial 

image data for each of the right and left eyes and the 

mouth; 

wherein said facial feature searching range setting means 
comprises mouth searching range setting means and 
facial contour searching range setting means which sets 
ranges in the x-axis and y-axis directions to include 
facial contours referenced to the points specified as the 
right and left eyes as the searching ranges for the facial 
contours in the areas at the eye height in the y-axis 
direction based on the average of the differences 
between the y-axis coordinate of the point specified as 
the mouth and the y-axis coordinates of the points 
specified as the right and left eyes, 
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the difference between the x-coordinate data of the 
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where the x-coordinate axis represents the side-to-side 
direction of the facial image and the y-coordinate 
axis represents the vertical direction of the facial 
image; and 

facial feature extracting means for extracting an area of 5 
each facial feature, including the mouth, within the 
searching range set by each corresponding facial fea- 
ture searching range setting means; 

wherein it is determined whether the extracted mouth area 
is correct by determining whether a line connecting 30 
right and left end points in the x-axis direction of the 
mouth area is greater than a predetermined angle rela- 
tive to a line through a particular point in each of the 
right and left eyes. 

18. A facial image processing apparatus according to 15 
claim 17, wherein 

eyes searching range setting means, and eyebrows search- 
ing range setting means are further disposed as the 
facial feature searching range setting means; 

a mouth area extracting means, eye area extracting means, 
eyebrow area extracting means, and facial contour 
extracting means are disposed as the facial feature 
extracting means; and 

the contours of the mouth, right and left eyes, eyebrows, ^ 
and the facial contour are extracted by the respective 
extracting means after the respective searching ranges 
are set by the respective searching range setting means 
based on the coordinate data of the points specified for 
the right and left eye and mouth. 3Q 

19. A facial image processing apparatus according to 
claim 17, wherein 

the mouth searching range setting means sets a range in 
the x-axis and y-axis directions including the mouth 
area referenced to the point specified as the mouth as 35 
the mouth searching range based on 
the difference between the x-coordinate data of the 
points specified as the right and left eyes. 

20. A facial image processing apparatus according to 
claim 17, wherein ^ 

the mouth searching range setting means sets a range in 
the x-axis and y-axis directions including the mouth 
area referenced to the point specified as the mouth as 
the mouth searching range based on 



45 



points specified as the right and left eyes, and 
the average of the differences between the y-axis coor- 
dinate of the point specified as the mouth and the 
y-axis coordinates of the points specified as the right 
and left eyes. 50 

21. A facial image processing apparatus according to 
claim 18, wherein 

the eyes searching range setting means sets a range in the 
x-axis and y-axis directions including the eye area 
referenced to the point specified as the eye as the eye 55 
searching range based on 

the difference between the x-axis coordinate data of the 
points specified as the right and left eyes. 

22. A facial image processing apparatus according to 
claim 18, wherein 60 

the eyebrows searching range setting means sets a range 
in the x-axis and y-axis directions including the eye- 
brow area referenced to the point specified as the 
corresponding eye as the eyebrow searching range 
based on 65 

the difference between the x-axis coordinate data of the 
points specified as the right and left eyes. 



23. A facial image processing apparatus according to 
claim 17, wherein 

the facial contour searching range setting means sets 
ranges in the x-axis and y-axis directions at the mouth 
height in the y-axis direction to include the facial 
contours in the mouth area referenced to the point 
specified as the mouth 

based on 

the average of the differences between the y-axis coor- 
dinate of the point specified as the mouth and the 
y-axis coordinates of the points specified as the right 
and left eyes. 

24. A facial image processing apparatus according to 
claim 18, wherein 

the facial contour searching range setting means sets a 
range in the x-axis and y-axis directions in the area of 
the mouth in the x-axis direction to include the facial 
contour in the mouth area referenced to the point 
specified as the mouth based on 
the average of the differences between the y-axis coor- 
dinate of the point specified as the mouth and the 
y-axis coordinates of the points specified as the right 
and left eyes. 

25. A facial image processing apparatus for extracting 
facial features and outputting corresponding coordinate data 
based on facial image data comprises: 

facial feature searching range setting means for setting, 
for each facial feature to be extracted, a searching range 
based on a plurality of specific points including contour 
points and referenced to a particular one of the plurality 
of the specific points specified for that facial feature, 
said specific points being pre-defined in the facial 

image data for at least the right and left eyes and the 

mouth; 

facial feature extracting means for extracting an area of 
each facial feature, including the mouth, within the 
searching range set by each corresponding facial fea- 
ture searching range setting means; 
wherein said searching range setting means sets ranges 
for facial contours in areas outside of the eyes and in 
an area on at least one side of the mouth to include 
said facial contours referenced to the points specified 
as the right and left eyes based on the average of the 
differences between the y-axis coordinate of the 
point specified as the mouth and y-axis coordinates 
of the points specified as the right and left eyes, 
where the x-coordinate axis represents the side-to-side 
direction of the facial image and the y-coordinate 
axis represents the vertical direction of the facial 
image; 

contour point extracting means for extracting the coordi- 
nate data of contour points forming the contour of each 
facial feature based on the respective areas obtained by 
the facial feature extracting means; and 

extracted content evaluating means for evaluating, based 
on the coordinate values of a specific point in the area 
of each facial feature, the relative positions of the 
extracted feature images to determine whether said 
facial features were correctly extracted, wherein the 
evaluation process for determining correct extraction of 
the mouth area includes determining whether a line 
connecting right and left end points in the x-axis 
direction of the mouth area is greater than a predeter- 
mined angle relative to a line through a particular point 
in each of the right and left eyes. 

26. A facial image processing apparatus for extracting 
facial features and outputting the corresponding coordinate 
data based on facial image data comprising: 
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facial feature searching range setting means for setting, 
for each facial feature to be extracted, a searching range 
based on a plurality of specific points including contour 
points and referenced to a particular one of the plurality 
of the specific points specified for that facial feature, 
said specific points being pre-defined in the facial 
image data for at least the right and left eyes and the 
mouth; 

facial feature extracting means for extracting an area of 
each facial feature, including the mouth, within the 
searching range set by each corresponding facial fea- 
ture searching range setting means; 

contour point extracting means for extracting the coordi- 
nate data of contour points forming the contour of each 
facial feature based on the respective areas obtained by 
the facial feature extracting means; and 

extracted content evaluating means for evaluating, based 
on the coordinate values of a specific point in the area 
of each facial feature, the relative positions of the 
extracted feature images to determine whether said 
facial features were correctly extracted; wherein 

the extracted content evaluating means, in the evaluation 
process for determining correct extraction of the mouth 
area extracted by the mouth area extracting means, 
compares the y-axis coordinates of the right and left 
end points in the x-axis direction of the extracted 
mouth area with the y-axis coordinate of the mid- 
point in the x-axis direction at the top edge of the 
extracted mouth area 

where the x-coordinate axis represents the side-to- 
side direction of the facial image and the 
y-coordinate axis represents the vertical direction 
of the facial image; and 
determines the extracted mouth area to represent an 
open mouth when the y-axis coordinate of the mid- 
point in the x-axis direction is smaller than the y-axis 
coordinate of either right or left end point in the 
x-axis direction of said mouth area, 
determines the extracted mouth to be the lower lip 
only, and 

sets a searching range for extracting the upper lip. 

27. A facial image processing apparatus according to 
claim 25, wherein 

the extracted content evaluating means, when the 
extracted content is an eye or the mouth, evaluates 
correct extraction of the eye or mouth by determining 
whether the respective points specified for the eyes or 
mouth are contained within the extracted areas. 

28. A facial image processing apparatus according to 
claim 25, wherein 

the extracted content evaluating means, when the 
extracted content is an eyebrow area, evaluates correct 
extraction of the eyebrow area by evaluating the posi- 
tions of the coordinates of the right and left end points 
of the extracted eyebrow area relative to the coordi- 
nates of the point specified as the corresponding eye. 

29. A facial image processing apparatus for extracting 
facial features and outputting the corresponding coordinate 
data based on facial image data comprises: 

facial feature extracting means for extracting the area of 
each facial feature using a coordinate system wherein 
the x-coordinate axis represents the side-to-side direc- 
tion of the facial image and the y-coordinate axis 
represents the vertical direction of the facial image; and 

facial feature contour point extracting means for obtaining 
the contour points forming the contour of the eyes or 
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mouth in the facial feature area extracted by the facial 
feature extracting means, and 

then obtaining from said contour points the coordinates 
of the right and left end points in the x-axis direction 
of said facial feature area, 

then setting, in the area near the midpoint between said 
right and left end points, a searching range with a 
width in the x-axis direction equal to a known M> 
fraction of the distance between the right and left end 
points, and 

obtaining within said searching range the coordinates 
of the top and bottom end points in the y-axis 
direction of the eye or mouth area. 

30. A facial image processing apparatus according to 
claim 29, wherein: 

the coordinates of the contour points between the obtained 
four right, left, top, and bottom end points are obtained 
with the x-axis coordinate thereof calculated as the 
average of the x-axis coordinates of the contour points 
on either side of the contour point being calculated. 

31. A facial image processing apparatus for extracting 
facial features and outputting the corresponding coordinate 
data based on facial image data comprising: 

facial feature extracting means for extracting the area of 

each facial feature; 
contour point extracting means for obtaining the contour 

points forming the contour of the facial feature based 

on the area extracted by the facial feature extracting 

means; and 

extracted content evaluating means for evaluating the 
relative positions of the extracted areas based on the 
coordinate values of a specific point in each facial 
feature area to determine whether the facial feature 
areas were correctly extracted; wherein: 
the extracted content evaluating means, when the 
extracted feature is the mouth, determines whether 
the extracted area is correct by determining whether 
a line connecting the right and left end points in the 
x-axis direction of the mouth area is sloped greater 
than a predetermined angle relative to a line through 
a specific point in each of the right and left eyes, 
where the x-coordinate axis represents the side-to- 
side direction of the facial image and the 
y-coordinate axis represents the vertical direction 
of the facial image. 

32. A facial image processing apparatus for extracting 
facial features and outputting the corresponding coordinate 
data based on facial image data comprising: 

facial feature extracting means for extracting the area of 

each facial feature; 
contour point extracting means for obtaining the contour 

points forming the contour of the facial feature based 

on the area extracted by the facial feature extracting 

means; and 

extracted content evaluating means for evaluating the rela- 
tive positions of the extracted areas based on the coordinate 
values of a specific point in each facial feature area to 
determine whether the facial feature areas were correctly 
extracted; wherein: 

the extracted content evaluating means, when the 
extracted area is a facial contour area, determines 
whether the extracted area is correct by 
processing the plural contour points forming the facial 
contour in units of three adjacent contour points, and 
comparing the coordinates of the middle of each three 
contour points with the average of the coordinates of 
the two end contour points, and 
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substitutes the average of the coordinates of the two end 
contour points for the coordinates of the middle contour 
point if the position of the middle contour point is 
determined to be to the inside of the face relative to the 
adjacent contour points, 5 
where the x-coordinate axis represents the side-to-side 
direction of the facial image and the y-coordinate 
axis represents the vertical direction of the facial 
image. 

33. A computer implemented image processing method, 10 
comprising the steps of: 

scanning and digitizing an image into a first portion of 
computer memory; 

identifying a plurality of specific feature points present in 
the digitized image, wherein each identified specific 15 
feature point corresponds to at least one distinct image 
feature, including right and left eyes and a mouth, 
present within the digitized image; 

determining an image searching range for each image ^ Q 
feature, including the mouth, facial contours horizon- 
tally adjacent the mouth and eyes, based on the iden- 
tified specific feature points and a predetermined set of 
feature-classification dependent rules, wherein the 
image searching range for said facial contours horizon- ^ 
tally adjacent the mouth and eyes is determined based 
on a difference between vertical positions of the spe- 
cific feature point for each eye and the mouth; 

extracting feature characteristics corresponding to each 
image feature, including the mouth, by evaluating 30 
digitized image portions defined by the determined 
source searching ranges; 

determining whether the extracted mouth is correct by 
determining whether a line connecting right and left 
end points in the x-axis direction of the mouth is greater 35 
than a predetermined angle relative to a line through a 
particular point in each of the right and left eyes; and 

storing the extracted feature characteristics as coordinate 
data within a second portion of the computer memory. 

34. The method of claim 33, wherein the image source 40 
comprises a facial image and the image features further 
include eyebrows. 

35. The method of claim 34, wherein said feature point 
identification step comprises: 

displaying the digitized facial image to a user; and 45 
capturing user input based on the displayed image, the 
user input comprising relative position coordinates of 
the mouth and eyes of the digitized facial image. 

36. The method of claim 33, wherein the image searching 
range determining step for the mouth feature comprises: 50 

calculating a difference between horizontal positions of 
the specific feature point for each eye; and 

determining the image searching range for the mouth 
based on the calculated difference. 55 

37. The method of claim 33, wherein said image searching 
range determining step for the mouth comprises: 

calculating a first difference between horizontal positions 

of the specific feature point for each eye; 
calculating a second difference between vertical positions go 

of the specific feature points for the mouth and each 

eye; and 

determining the image searching range for the mouth 
based on the first and second calculated differences and 
the specific feature point for the mouth. 65 

38. The method of claim 33, wherein the image searching 
range determining step for a given eye comprises: 
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calculating a difference between horizontal positions of 
the specific feature point for each eye; and 

determining the image searching range for the given eye 
based on the calculated difference and the specific 
feature point of the given eye. 

39. The method of claim 34, wherein the image searching 
range determining step for a given eyebrow comprises: 

calculating a difference between horizontal positions of 
the specific feature point for each eye; and 

determining the image searching range for the given 
eyebrow based on the calculated difference and the 
specific feature point of the eye positioned closest to 
the given eyebrow. 

40. The method of claim 34, further comprising evaluat- 
ing the coordinate data for at least one feature against the 
corresponding specific feature point to determine whether 
the feature was correctly extracted. 

41. A computer implemented image processing method, 
comprising the steps of: 

scanning and digitizing a facial image into a first portion 
of computer memory; 

identifying a plurality of specific feature points present in 
the digitized facial image, wherein each identified 
specific feature point corresponds to at least one dis- 
tinct facial image feature, including at least a mouth, 
present within the digitized image; 

determining an image searching range for each facial 
image feature based on the identified specific feature 
points and a predetermined set of feature-classification 
dependent rules; 

extracting feature characteristics corresponding to each 
image feature by evaluating digitized image portions 
defined by the determined source searching ranges; 

storing the extracted feature characteristics as coordinate 
data within a second portion of the computer memory; 
and 

evaluating the coordinate data for at least one facial 
feature against the corresponding specific feature point 
to determine whether the feature was correctly 
extracted, wherein the coordinate data evaluating step 
comprises: 

retrieving coordinate data corresponding to the mouth 
of the digitized facial image; 

identifying mouth comer points within the retrieved 
coordinate data; 

calculating a lip cusp based on the identified mouth 
comer points; 

determining whether the lip cusp is positioned under- 
neath either of the identified mouth corner points; 
and 

setting an additional searching range for extracting an 
upper lip portion of the mouth when it is determined 
in said lip cusp positioning determining step that the 
lip cusp is positioned beneath one of the identified 
mouth corner points. 

42. A computer implemented image processing method, 
comprising the steps of: 

scanning and digitizing a facial image into a first portion 
of computer memory; 

identifying a plurality of specific feature points present in 
the digitized facial image, wherein each identified 
specific feature point corresponds to at least one dis- 
tinct facial image feature, including at least right and 
left eyes and a mouth, present within the digitized 
image; 
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determining an image searching range for each facial a line connecting right and left end points in the x-axis 

image feature, including the mouth, based on the iden- direction of the mouth is greater than a predetermined 

tified specific feature points and a predetermined set of angle relative to a line through a particular point in each 

feature-classification dependent rules; of the right and left eyes; and 

extracting feature characteristics corresponding to each 5 computer readable program code for storing the extracted 
image feature, including the mouth, by evaluating feature characteristics as coordinate data within corn- 
digitized image portions defined by the determined P^ T memorv - , . 
source searching ranges; 46 - ™ e arUcle of manufacture of claim 45, wherein the 
, r x . . ,. A image source comprises a facial image and the image 

storing the extracted feature characteristics as coordinate brows 

data within a second portion of the computer memory; 4? ^ of mamlfacture of claim 46> wherein ^ 

computer readable specific feature points identification code 

evaluating the coordinate data for at least one facial comprises: 

feature against the corresponding specific feature point com puter readable program code for displaying the digi- 

to determine whether the feature was correctly J5 ^ fedal . e tQ a ^ and 

extracted, wherein the coordinate data evaluating step . , , « , r . . 

qauovlw, wiiviviu uiv W & v computer readable program code for capturing user input 

comprises. based on the displayed image, the user input compris- 

retrieving coordinate data corresponding to the mouth relative n of the mouth ^ 

of the digitized fac,al ^.mage; of the digitized facial image, 

determimng whether me retrieved coordinate data 2Q ^ ^ ^ rf manufaclu * of cIaim 45 wherein ^ 

includes the specific feature point for the mouth; and utef jm determining code, 

determining whether a line connectmg right and left fof ^ kam ^ ^ 

end points in the x-axis direction of the mouth is , , t r , . , . . 

greater than a predetermined angle relative to a line com P ut " readab1 ^ P^S™? for plating » dl f r - 

Through a particular point in each of the right and left „ cnce ****** horizontal positions of the specific fea- 

° "25 ture point for each eye; and 

43. TOe method of claim 40, wherein the coordinate data computer readable program code for determining the 
evaluating step comprises: """B 6 ^hing range for the mouth based on the 

... ,. . , . .. . , calculated difference, 

retrieving coordmate data corresponding to a given eye of ^ ^ ^ of manufacture of claim ^ wherein ^ 

the digitized facial image; and 30 computer readabl e ima ge searching range determining code, 

determining whether the retrieved coordinate data for the mouth feature, comprises: 

includes the specific feature point for the given eye. compu ter readable program code for calculating a first 

44. The method of claim 40, wherein the coordinate data difference between horizontal positions of the specific 
evaluating step comprises: fcature poim for each eye; 

retrieving coordinate data corresponding to a given eye- 35 computer rea dable program code for calculating a second 

brow of the digitized facial image; and difference between vertical positions of the specific 

determining whether the retrieved coordinate data is feature points for the mouth and each eye; and 

within a predetermined distance from the specific fea- computer readable program code for determining the 

ture point for an eye closest to the given eyebrow. i mage searching range for the mouth based on the first 

45. An article of manufacture, comprising a computer and second calculated differences and the specific fea- 
usable medium having readable code embodied therein for ture point for the mouth. 

processing a digitized image, the computer readable pro- 50, The article of manufacture of claim 45, wherein said 

gram code comprising: computer readable image searching range determining code, 

computer readable program code for identifying a plural- 45 for a given eye feature, comprises: 

ity of specific feature points present in the digitized computer readable program code for calculating a differ- 

image, wherein each identified specific feature point ence between horizontal positions of the specific fea- 

corresponds to at least one distinct image feature, ture point for each eye; and 

including right and left eyes and a mouth, present computer readable program code for determining the 

within the digitized image; 5Q searching range for the given eye based on the 

computer readable program code for determining an calculated difference and the specific feature point of 

image searching range for each image feature, includ- the given eye. 

ing a mouth and facial contours horizontally adjacent 51. The article of manufacture of claim 46, wherein said 

the mouth and eyes, based on the identified specific computer readable image searching range determining code, 

feature points and a predetermined set of feature- 55 for a given eyebrow feature, comprises: 

classification dependent rules, wherein the image range computer readable program code for calculating a differ- 

for said facial contours horizontally adjacent the mouth ence between horizontal positions of the specific fea- 

and eyes is determined based on a difference between ture point for each eye; and 

vertical positions of the specific feature point for each computer readable program code for determining the 

eye and the mouth; 60 image searching range for the given eyebrow based on 

computer readable program code for extracting feature the calculated difference and the specific feature point 
characteristics corresponding to each image feature, 0 f the eye positioned closest to the given eyebrow, 
including the mouth, by evaluating digitized image 52. The article of manufacture of claim 46, further corn- 
portions defined by the determined source searching prising computer readable program code for evaluating the 
ranges; 6 5 coordinate data for at least one feature against the corre- 

computer readable program code for determining whether sponding specific feature point to determine whether the 

the extracted mouth is correct by determining whether feature was correctly extracted. 
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53. An article of manufacture, comprising a computer 
usable medium having readable code embodied therein for 
processing a digitized image, the computer readable pro- 
gram code comprising: 

computer readable program code for identifying a plural- 
ity of specific feature points present in the digitized 
image, wherein each identified specific feature point 
corresponds to at least one distinct image feature, 
including at least a mouth, present within the digitized 
image; 

computer readable program code for determining an 
image searching range for each image feature based on 
the identified specific feature points and a predeter- 
mined set of feature-classification dependent rules; 

computer readable program code for extracting feature 
characteristics corresponding to each image feature by 
evaluating digitized image portions defined by the 
determined source searching ranges; 

computer readable program code for storing the extracted 
feature characteristics as coordinate data within com- 
puter memory: and 

computer readable code for evaluating the coordinate data 
for at least one feature against the corresponding spe- 
cific feature point to determine whether the feature was 
correctly extracted, wherein said computer readable 
coordinate data evaluating code comprises: 
computer readable program code for retrieving coordi- 
nate data corresponding to the mouth of the digitized 
facial image; 

computer readable program code for identifying mouth 
corner points within the retrieved coordinate data; 

computer readable program code for calculating a lip 
cusp based on the identified mouth corner points; 

computer readable program code for determining 
whether the lip cusp is positioned underneath either 
of the identified mouth corner points; and 
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computer readable program code for setting an addi- 
tional searching range for extracting an upper lip 
portion of the mouth when it is determined by said 
lip cusp positioning determining means that the lip 
cusp is positioned beneath one of the identified 
mouth corner points. 

54. The article of manufacture of claim 52, wherein said 
computer readable coordinate data evaluating code com- 
prises: 

computer readable program code for retrieving coordinate 
data corresponding to the mouth of the digitized facial 
image; and 

computer readable program code for determining whether 
the retrieved coordinate data includes the specific fea- 
ture point for the mouth. 

55. The article of manufacture of claim 52, wherein said 
computer readable coordinate data evaluating code com- 
prises: 

computer readable program code for retrieving coordinate 
data corresponding to a given eye of the digitized facial 
image; and 

computer readable program code for determining whether 
the retrieved coordinate data includes the specific fea- 
ture point for the given eye. 

56. The article of manufacture of claim 52, wherein said 
computer readable coordinate data evaluating code com- 
prises: 

computer readable program code for retrieving coordinate 
data corresponding to a given eyebrow of the digitized 
facial image; and 

computer readable program code for determining whether 
the retrieved coordinate data is within a predetermined 
distance from the specific feature point for an eye 
closest to the given eyebrow. 
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